Extensible multi-tenant cloud-management system and methods for extending functionalities and services provided by a multi-tenant cloud-managment system

ABSTRACT

The current document is directed to an interface and authorization service that allows users of a cloud-director management subsystem of distributed, multi-tenant, virtual data centers to extend the services and functionalities provided by the cloud-director management subsystem. A cloud application programming interface (“API”) entrypoint represents a request/response RESTful interface to services and functionalities provided by the cloud-director management subsystem as well as to service extensions provided by users. The cloud API entrypoint includes a service-extension interface and an authorization-service management interface. The cloud-director management subsystem provides the authorization service to service extensions that allow the service extensions to obtain, from the authorization service, an indication of whether or not a request directed to the service extension through the cloud API entrypoint is authorized.

TECHNICAL FIELD

The current document is directed to virtualization technologies and, inparticular, to a subsystem of multi-tenant distributed cloud-computingfacilities and to methods carried out by the subsystem that provide foraccess-controlled extensions to services and functionalities provided bythe subsystem.

BACKGROUND

The development and evolution of modern computing has, in many ways,been facilitated by the power of logical abstraction. Early computerswere manually programmed by slow and tedious input of machineinstructions into the computers' memories. Over time, assembly-languageprograms and assemblers were developed in order to provide a level ofabstraction, namely assembly-language programs, above themachine-instruction hardware-interface level, to allow programmers tomore rapidly and accurately develop programs. Assembly-language-basedoperations are more easily encoded by human programmers thanmachine-instruction-based operations, and assemblers provided additionalfeatures, including assembly directives, routine calls, and a logicalframework for program development. The development of operating systemsprovided yet another type of abstraction that provided programmers withlogical, easy-to-understand system-call interfaces to computer-hardwarefunctionality. As operating systems developed, additional internallevels of abstraction were created within operating systems, includingvirtual memory, implemented by operating-system paging of memory pagesbetween electronic memory and mass-storage devices, which providedeasy-to-use, linear memory-address spaces much larger than could beprovided by the hardware memory of computer systems. Additional levelsof abstractions were created in the programming-language domain, withcompilers developed for a wide variety of compiled languages thatgreatly advanced the ease of programming and the number and capabilitiesof programming tools with respect those provided by assemblers andassembly languages. Higher-level scripting languages and special-purposeinterpreted languages provided even higher levels of abstraction andgreater ease of application development in particular areas. Similarly,block-based and sector-based interfaces to mass-storage devices havebeen abstracted through many levels of abstraction to modem databasemanagement systems, which provide for high-available and fault-tolerantstorage of structured data that can be analyzed, interpreted, andmanipulated through powerful high-level query languages.

In many ways, a modem computer system can be thought of as manydifferent levels of abstractions along many different, ofteninterdependent, dimensions. More recently, powerful new levels ofabstraction have been developed with respect to virtual machines, whichprovide virtual execution environments for application programs andoperating systems. Virtual-machine technology essentially abstracts thehardware resources and interfaces of a computer system on behalf of oneor multiple virtual machines, each comprising one or more applicationprograms and an operating system. Even more recently, the emergence ofcloud computing services can provide abstract interfaces to enormouscollections of geographically dispersed data centers, allowingcomputational service providers to develop and deploy complexInternet-based services that execute on tens or hundreds of physicalservers through abstract cloud-computing interfaces.

The abstract interfaces provided by cloud-computing allow users toaccess various services and functionalities used to manage distributed,multi-tenant, virtual data centers. Designers, developers, vendors, andusers of these abstract interfaces continue to seek functionality toexpand the services and functionalities accessed through the abstractinterfaces.

SUMMARY

The current document is directed to an interface and authorizationservice that allows users of a cloud-director management subsystem ofdistributed, multi-tenant, virtual data centers to extend the servicesand functionalities provided by the cloud-director management subsystem.A cloud application programming interface (“API”) entrypoint representsa request/response RESTful interface to services and functionalitiesprovided by the cloud-director management subsystem as well as toservice extensions provided by users. The cloud API entrypoint includesa service-extension interface and an authorization-service managementinterface. The cloud-director management subsystem provides theauthorization service to service extensions that allow the serviceextensions to obtain, from the authorization service, an indication ofwhether or not a request directed to the service extension through thecloud API entrypoint is authorized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a general architectural diagram for various types ofcomputers.

FIG. 2 illustrates an Internet-connected distributed computer system.

FIG. 3 illustrates cloud computing.

FIG. 4 illustrates generalized hardware and software components of ageneral-purpose computer system, such as a general-purpose computersystem having an architecture similar to that shown in FIG. 1.

FIG. 5 illustrates one type of virtual machine and virtual-machineexecution environment.

FIG. 6 illustrates an OVF package.

FIG. 7 illustrates virtual data centers provided as an abstraction ofunderlying physical-data-center hardware components.

FIG. 8 illustrates virtual-machine components of a virtual-data-centermanagement server and physical servers of a physical data center abovewhich a virtual-data-center interface is provided by thevirtual-data-center management server.

FIG. 9 illustrates a cloud-director level of abstraction.

FIG. 10 illustrates virtual-cloud-connector nodes (“VCC nodes”) and aVCC server, components of a distributed system that provides multi-cloudaggregation and that includes a cloud-connector server andcloud-connector nodes that cooperate to provide services that aredistributed across multiple clouds.

FIG. 11 illustrates the VCC server and VCC nodes in a slightly differentfashion than the VCC server and VCC nodes are illustrated in FIG. 10.

FIG. 12 illustrates one implementation of a VCC node.

FIG. 13 illustrates electronic communications between a client andserver computer.

FIG. 14 illustrates the role of resources in RESTful APIs.

FIGS. 15A-D illustrate four basic verbs, or operations, provided by theHTTP application-layer protocol used in RESTful applications.

FIGS. 16A-B illustrate extension of the simple RESTful API discussedabove with reference to FIGS. 14 and 15A-B.

FIG. 17 illustrates the use of additional links and responses in orderto inform a client of newly available resources as a result of extensionof a RESTful API.

FIGS. 18A-C illustrate the type of functionality present within a serverthat handles requests, including GET requests from clients.

FIGS. 19A-B illustrate an example link registry.

FIG. 20 illustrates a JSON encoding of a particular link-registry entry.

FIG. 21 shows example method calls to a simple link-registry API.

FIGS. 22A-C illustrate a recently developed approach that allows forextension of the facilities and services provided by the cloud directorfor management of distributed, multi-tenant VDCs.

FIG. 23 illustrates data objects maintained in an authorization-servicedatabase maintained by the cloud director to support the authorizationservice provided by the cloud-director backend to supportcloud-director-provided service extensions.

FIG. 24 provides a table-based description of a portion of theauthorization-service management interface.

FIG. 25 shows, as one example, an HTTP/XML implementation of the ASMIoperations 2402 in FIG. 24 associated with resource classes.

FIG. 26 shows the call interface to the authorization service providedby the cloud director.

FIGS. 27A-B provide control-flow diagrams that illustrate animplementation of the routine “authorize.”

FIG. 28 provides a control-flow diagram for the routine “authorizeservices.”

DETAILED DESCRIPTION OF EMBODIMENTS

As discussed above, modern computing can be considered to be acollection of many different levels of abstraction above the physicalcomputing-hardware level that includes physical computer systems,data-storage systems and devices, and communications networks. Thecurrent document is related to a multi-cloud-aggregation level ofabstraction that provides homogenous-cloud and heterogeneous-clouddistributed management services, each cloud generally an abstraction ofa large number of virtual resource pools comprising processing, storage,and network resources, each of which, in turn, can be considered to be acollection of abstractions above underlying physical hardware devices.The current document is directed to extension of services andfunctionalities provided by a management subsystem that provides amanagement interface to distributed, multi-tenant, virtual data centersresident within various physical cloud-computing facilities and, inparticular, to an authorization service that allows users of themanagement-subsystem interface to extend services in amulti-tenancy-compatible fashion. The following discussion includesthree subsections: (1) Computer Architecture, Virtualization, ElectronicCommunications, and Virtual Networks; (2) RESTful APIs; and (3)Authorization Service to Facilitate Multi-Tenancy in Cloud-ComputingFacilities.

Computer Architecture, Virtualization, Electronic Communications, andVirtual Networks

The term “abstraction” is not, in any way, intended to mean or suggestan abstract idea or concept. Computational abstractions are tangible,physical interfaces that are implemented, ultimately, using physicalcomputer hardware, data-storage devices, and communications systems.Instead, the term “abstraction” refers, in the current discussion, to alogical level of functionality encapsulated within one or more concrete,tangible, physically-implemented computer systems with definedinterfaces through which electronically-encoded data is exchanged,process execution launched, and electronic services are provided.Interfaces may include graphical and textual data displayed on physicaldisplay devices as well as computer programs and routines that controlphysical computer processors to carry out various tasks and operationsand that are invoked through electronically implemented applicationprogramming interfaces (“APIs”) and other electronically implementedinterfaces. There is a tendency among those unfamiliar with moderntechnology and science to misinterpret the terms “abstract” and“abstraction,” when used to describe certain aspects of moderncomputing. For example, one frequently encounters assertions that,because a computational system is described in terms of abstractions,functional layers, and interfaces, the computational system is somehowdifferent from a physical machine or device. Such allegations areunfounded. One only needs to disconnect a computer system or group ofcomputer systems from their respective power supplies to appreciate thephysical, machine nature of complex computer technologies. One alsofrequently encounters statements that characterize a computationaltechnology as being “only software,” and thus not a machine or device.Software is essentially a sequence of encoded symbols, such as aprintout of a computer program or digitally encoded computerinstructions sequentially stored in a file on an optical disk or withinan electromechanical mass-storage device. Software alone can do nothing.It is only when encoded computer instructions are loaded into anelectronic memory within a computer system and executed on a physicalprocessor that so-called “software implemented” functionality isprovided. The digitally encoded computer instructions are an essentialcontrol component of processor-controlled machines and devices, no lessessential than a cam-shaft control system in an internal-combustionengine. Multi-cloud aggregations, cloud-computing services,virtual-machine containers and virtual machines, communicationsinterfaces, and many of the other topics discussed below are tangible,physical components of physical, electro-optical-mechanical computersystems.

FIG. 1 provides a general architectural diagram for various types ofcomputers. The computer system contains one or multiple centralprocessing units (“CPUs”) 102-105, one or more electronic memories 108interconnected with the CPUs by a CPU/memory-subsystem bus 110 ormultiple busses, a first bridge 112 that interconnects theCPU/memory-subsystem bus 110 with additional busses 114 and 116, orother types of high-speed interconnection media, including multiple,high-speed serial interconnects. These busses or serialinterconnections, in turn, connect the CPUs and memory with specializedprocessors, such as a graphics processor 118, and with one or moreadditional bridges 120, which are interconnected with high-speed seriallinks or with multiple controllers 122-127, such as controller 127, thatprovide access to various different types of mass-storage devices 128,electronic displays, input devices, and other such components,subcomponents, and computational resources. It should be noted thatcomputer-readable data-storage devices include optical andelectromagnetic disks, electronic memories, and other physicaldata-storage devices. Those familiar with modern science and technologyappreciate that electromagnetic radiation and propagating signals do notstore data for subsequent retrieval, and can transiently “store” only abyte or less of information per mile, far less information than neededto encode even the simplest of routines.

Of course, there are many different types of computer-systemarchitectures that differ from one another in the number of differentmemories, including different types of hierarchical cache memories, thenumber of processors and the connectivity of the processors with othersystem components, the number of internal communications busses andserial links, and in many other ways. However, computer systemsgenerally execute stored programs by fetching instructions from memoryand executing the instructions in one or more processors. Computersystems include general-purpose computer systems, such as personalcomputers (“PCs”), various types of servers and workstations, andhigher-end mainframe computers, but may also include a plethora ofvarious types of special-purpose computing devices, includingdata-storage systems, communications routers, network nodes, tabletcomputers, and mobile telephones.

FIG. 2 illustrates an Internet-connected distributed computer system. Ascommunications and networking technologies have evolved in capabilityand accessibility, and as the computational bandwidths, data-storagecapacities, and other capabilities and capacities of various types ofcomputer systems have steadily and rapidly increased, much of moderncomputing now generally involves large distributed systems and computersinterconnected by local networks, wide-area networks, wirelesscommunications, and the Internet. FIG. 2 shows a typical distributedsystem in which a large number of PCs 202-205, a high-end distributedmainframe system 210 with a large data-storage system 212, and a largecomputer center 214 with large numbers of rack-mounted servers or bladeservers all interconnected through various communications and networkingsystems that together comprise the Internet 216. Such distributedcomputing systems provide diverse arrays of functionalities. Forexample, a PC user sitting in a home office may access hundreds ofmillions of different web sites provided by hundreds of thousands ofdifferent web servers throughout the world and may accesshigh-computational-bandwidth computing services from remote computerfacilities for running complex computational tasks.

Until recently, computational services were generally provided bycomputer systems and data centers purchased, configured, managed, andmaintained by service-provider organizations. For example, an e-commerceretailer generally purchased, configured, managed, and maintained a datacenter including numerous web servers, back-end computer systems, anddata-storage systems for serving web pages to remote customers,receiving orders through the web-page interface, processing the orders,tracking completed orders, and other myriad different tasks associatedwith an e-commerce enterprise.

FIG. 3 illustrates cloud computing. In the recently developedcloud-computing paradigm, computing cycles and data-storage facilitiesare provided to organizations and individuals by cloud-computingproviders. In addition, larger organizations may elect to establishprivate cloud-computing facilities in addition to, or instead of,subscribing to computing services provided by public cloud-computingservice providers. In FIG. 3, a system administrator for anorganization, using a PC 302, accesses the organization's private cloud304 through a local network 306 and private-cloud interface 308 and alsoaccesses, through the Internet 310, a public cloud 312 through apublic-cloud services interface 314. The administrator can, in eitherthe case of the private cloud 304 or public cloud 312, configure virtualcomputer systems and even entire virtual data centers and launchexecution of application programs on the virtual computer systems andvirtual data centers in order to carry out any of many different typesof computational tasks. As one example, a small organization mayconfigure and run a virtual data center within a public cloud thatexecutes web servers to provide an e-commerce interface through thepublic cloud to remote customers of the organization, such as a userviewing the organization's e-commerce web pages on a remote user system316.

Cloud-computing facilities are intended to provide computationalbandwidth and data-storage services much as utility companies provideelectrical power and water to consumers. Cloud computing providesenormous advantages to small organizations without the resources topurchase, manage, and maintain in-house data centers. Such organizationscan dynamically add and delete virtual computer systems from theirvirtual data centers within public clouds in order to trackcomputational-bandwidth and data-storage needs, rather than purchasingsufficient computer systems within a physical data center to handle peakcomputational-bandwidth and data-storage demands. Moreover, smallorganizations can completely avoid the overhead of maintaining andmanaging physical computer systems, including hiring and periodicallyretraining information-technology specialists and continuously payingfor operating-system and database-management-system upgrades.Furthermore, cloud-computing interfaces allow for easy andstraightforward configuration of virtual computing facilities,flexibility in the types of applications and operating systems that canbe configured, and other functionalities that are useful even for ownersand administrators of private cloud-computing facilities used by asingle organization.

FIG. 4 illustrates generalized hardware and software components of ageneral-purpose computer system, such as a general-purpose computersystem having an architecture similar to that shown in FIG. 1. Thecomputer system 400 is often considered to include three fundamentallayers: (1) a hardware layer or level 402; (2) an operating-system layeror level 404; and (3) an application-program layer or level 406. Thehardware layer 402 includes one or more processors 408, system memory410, various different types of input-output (“I/O”) devices 410 and412, and mass-storage devices 414. Of course, the hardware level alsoincludes many other components, including power supplies, internalcommunications links and busses, specialized integrated circuits, manydifferent types of processor-controlled or microprocessor-controlledperipheral devices and controllers, and many other components. Theoperating system 404 interfaces to the hardware level 402 through alow-level operating system and hardware interface 416 generallycomprising a set of non-privileged computer instructions 418, a set ofprivileged computer instructions 420, a set of non-privileged registersand memory addresses 422, and a set of privileged registers and memoryaddresses 424. In general, the operating system exposes non-privilegedinstructions, non-privileged registers, and non-privileged memoryaddresses 426 and a system-call interface 428 as an operating-systeminterface 430 to application programs 432-436 that execute within anexecution environment provided to the application programs by theoperating system. The operating system, alone, accesses the privilegedinstructions, privileged registers, and privileged memory addresses. Byreserving access to privileged instructions, privileged registers, andprivileged memory addresses, the operating system can ensure thatapplication programs and other higher-level computational entitiescannot interfere with one another's execution and cannot change theoverall state of the computer system in ways that could deleteriouslyimpact system operation. The operating system includes many internalcomponents and modules, including a scheduler 442, memory management444, a file system 446, device drivers 448, and many other componentsand modules. To a certain degree, modern operating systems providenumerous levels of abstraction above the hardware level, includingvirtual memory, which provides to each application program and othercomputational entities a separate, large, linear memory-address spacethat is mapped by the operating system to various electronic memoriesand mass-storage devices. The scheduler orchestrates interleavedexecution of various different application programs and higher-levelcomputational entities, providing to each application program a virtual,stand-alone system devoted entirely to the application program. From theapplication program's standpoint, the application program executescontinuously without concern for the need to share processor resourcesand other system resources with other application programs andhigher-level computational entities. The device drivers abstract detailsof hardware-component operation, allowing application programs to employthe system-call interface for transmitting and receiving data to andfrom communications networks, mass-storage devices, and other I/Odevices and subsystems. The file system 436 facilitates abstraction ofmass-storage-device and memory resources as a high-level,easy-to-access, file-system interface. Thus, the development andevolution of the operating system has resulted in the generation of atype of multi-faceted virtual execution environment for applicationprograms and other higher-level computational entities.

While the execution environments provided by operating systems haveproved to be an enormously successful level of abstraction withincomputer systems, the operating-system-provided level of abstraction isnonetheless associated with difficulties and challenges for developersand users of application programs and other higher-level computationalentities. One difficulty arises from the fact that there are manydifferent operating systems that run within various different types ofcomputer hardware. In many cases, popular application programs andcomputational systems are developed to run on only a subset of theavailable operating systems, and can therefore be executed within only asubset of the various different types of computer systems on which theoperating systems are designed to run. Often, even when an applicationprogram or other computational system is ported to additional operatingsystems, the application program or other computational system cannonetheless run more efficiently on the operating systems for which theapplication program or other computational system was originallytargeted. Another difficulty arises from the increasingly distributednature of computer systems. Although distributed operating systems arethe subject of considerable research and development efforts, many ofthe popular operating systems are designed primarily for execution on asingle computer system. In many cases, it is difficult to moveapplication programs, in real time, between the different computersystems of a distributed computer system for high-availability,fault-tolerance, and load-balancing purposes. The problems are evengreater in heterogeneous distributed computer systems which includedifferent types of hardware and devices running different types ofoperating systems. Operating systems continue to evolve, as a result ofwhich certain older application programs and other computationalentities may be incompatible with more recent versions of operatingsystems for which they are targeted, creating compatibility issues thatare particularly difficult to manage in large distributed systems.

For all of these reasons, a higher level of abstraction, referred to asthe “virtual machine,” has been developed and evolved to furtherabstract computer hardware in order to address many difficulties andchallenges associated with traditional computing systems, including thecompatibility issues discussed above. FIG. 5 illustrates one type ofvirtual machine and virtual-machine execution environment. FIG. 5 usesthe same illustration conventions as used in FIG. 4. In particular, thecomputer system 500 in FIG. 5 includes the same hardware layer 502 asthe hardware layer 402 shown in FIG. 4. However, rather than providingan operating system layer directly above the hardware layer, as in FIG.4, the virtualized computing environment illustrated in FIG. 5 featuresa virtualization layer 504 that interfaces through avirtualization-layer/hardware-layer interface 506, equivalent tointerface 416 in FIG. 4, to the hardware. The virtualization layerprovides a hardware-like interface 508 to a number of virtual machines,such as virtual machine 510, executing above the virtualization layer ina virtual-machine layer 512. Each virtual machine includes one or moreapplication programs or other higher-level computational entitiespackaged together with an operating system, such as application 514 andoperating system 516 packaged together within virtual machine 510. Eachvirtual machine is thus equivalent to the operating-system layer 404 andapplication-program layer 406 in the general-purpose computer systemshown in FIG. 4. Each operating system within a virtual machineinterfaces to the virtualization-layer interface 508 rather than to theactual hardware interface 506. The virtualization layer partitionshardware resources into abstract virtual-hardware layers to which eachoperating system within a virtual machine interfaces. The operatingsystems within the virtual machines, in general, are unaware of thevirtualization layer and operate as if they were directly accessing atrue hardware interface. The virtualization layer ensures that each ofthe virtual machines currently executing within the virtual environmentreceive a fair allocation of underlying hardware resources and that allvirtual machines receive sufficient resources to progress in execution.The virtualization-layer interface 508 may differ for differentoperating systems. For example, the virtualization layer is generallyable to provide virtual hardware interfaces for a variety of differenttypes of computer hardware. This allows, as one example, a virtualmachine that includes an operating system designed for a particularcomputer architecture to run on hardware of a different architecture.The number of virtual machines need not be equal to the number ofphysical processors or even a multiple of the number of processors. Thevirtualization layer includes a virtual-machine-monitor module 518 thatvirtualizes physical processors in the hardware layer to create virtualprocessors on which each of the virtual machines executes. For executionefficiency, the virtualization layer attempts to allow virtual machinesto directly execute non-privileged instructions and to directly accessnon-privileged registers and memory. However, when the operating systemwithin a virtual machine accesses virtual privileged instructions,virtual privileged registers, and virtual privileged memory through thevirtualization-layer interface 508, the accesses may result in executionof virtualization-layer code to simulate or emulate the privilegedresources. The virtualization layer additionally includes a kernelmodule 520 that manages memory, communications, and data-storage machineresources on behalf of executing virtual machines. The kernel, forexample, may maintain shadow page tables on each virtual machine so thathardware-level virtual-memory facilities can be used to process memoryaccesses. The kernel may additionally include routines that implementvirtual communications and data-storage devices as well as devicedrivers that directly control the operation of underlying hardwarecommunications and data-storage devices. Similarly, the kernelvirtualizes various other types of I/O devices, including keyboards,optical-disk drives, and other such devices. The virtualization layeressentially schedules execution of virtual machines much like anoperating system schedules execution of application programs, so thatthe virtual machines each execute within a complete and fully functionalvirtual hardware layer.

A virtual machine or virtual application, described below, isencapsulated within a data package for transmission, distribution, andloading into a virtual-execution environment. One public standard forvirtual-machine encapsulation is referred to as the “open virtualizationformat” (“OVF”). The OVF standard specifies a format for digitallyencoding a virtual machine within one or more data files. FIG. 6illustrates an OVF package. An OVF package 602 includes an OVFdescriptor 604, an OVF manifest 606, an OVF certificate 608, one or moredisk-image files 610-611, and one or more resource files 612-614. TheOVF package can be encoded and stored as a single file or as a set offiles. The OVF descriptor 604 is an XML document 620 that includes ahierarchical set of elements, each demarcated by a beginning tag and anending tag. The outermost, or highest-level, element is the envelopeelement, demarcated by tags 622 and 623. The next-level element includesa reference element 626 that includes references to all files that arepart of the OVF package, a disk section 628 that contains metainformation about all of the virtual disks included in the OVF package,a networks section 630 that includes meta information about all of thelogical networks included in the OVF package, and a collection ofvirtual-machine configurations 632 which further includes hardwaredescriptions of each virtual machine 634. There are many additionalhierarchical levels and elements within a typical OVF descriptor. TheOVF descriptor is thus a self-describing, XML file that describes thecontents of an OVF package. The OVF manifest 606 is a list ofcryptographic-hash-function-generated digests 636 of the entire OVFpackage and of the various components of the OVF package. The OVFcertificate 608 is an authentication certificate 640 that includes adigest of the manifest and that is cryptographically signed. Disk imagefiles, such as disk image file 610, are digital encodings of thecontents of virtual disks and resource files 612 are digitally encodedcontent, such as operating-system images. A virtual machine or acollection of virtual machines encapsulated together within a virtualapplication can thus be digitally encoded as one or more files within anOVF package that can be transmitted, distributed, and loaded usingwell-known tools for transmitting, distributing, and loading files. Avirtual appliance is a software service that is delivered as a completesoftware stack installed within one or more virtual machines that isencoded within an OVF package.

The advent of virtual machines and virtual environments has alleviatedmany of the difficulties and challenges associated with traditionalgeneral-purpose computing. Machine and operating-system dependencies canbe significantly reduced or entirely eliminated by packagingapplications and operating systems together as virtual machines andvirtual appliances that execute within virtual environments provided byvirtualization layers running on many different types of computerhardware. A next level of abstraction, referred to as virtual datacenters or virtual infrastructure, provide a data-center interface tovirtual data centers computationally constructed within physical datacenters. FIG. 7 illustrates virtual data centers provided as anabstraction of underlying physical-data-center hardware components. InFIG. 7, a physical data center 702 is shown below a virtual-interfaceplane 704. The physical data center consists of a virtual-data-centermanagement server 706 and any of various different computers, such asPCs 708, on which a virtual-data-center management interface may bedisplayed to system administrators and other users. The physical datacenter additionally includes generally large numbers of servercomputers, such as server computer 710, that are coupled together bylocal area networks, such as local area network 712 that directlyinterconnects server computer 710 and 714-720 and a mass-storage array722. The physical data center shown in FIG. 7 includes three local areanetworks 712, 724, and 726 that each directly interconnects a bank ofeight servers and a mass-storage array. The individual server computers,such as server computer 710, each includes a virtualization layer andruns multiple virtual machines. Different physical data centers mayinclude many different types of computers, networks, data-storagesystems and devices connected according to many different types ofconnection topologies. The virtual-data-center abstraction layer 704, alogical abstraction layer shown by a plane in FIG. 7, abstracts thephysical data center to a virtual data center comprising one or moreresource pools, such as resource pools 730-732, one or more virtual datastores, such as virtual data stores 734-736, and one or more virtualnetworks. In certain implementations, the resource pools abstract banksof physical servers directly interconnected by a local area network.

The virtual-data-center management interface allows provisioning andlaunching of virtual machines with respect to resource pools, virtualdata stores, and virtual networks, so that virtual-data-centeradministrators need not be concerned with the identities ofphysical-data-center components used to execute particular virtualmachines. Furthermore, the virtual-data-center management serverincludes functionality to migrate running virtual machines from onephysical server to another in order to optimally or near optimallymanage resource allocation, provide fault tolerance, and highavailability by migrating virtual machines to most effectively utilizeunderlying physical hardware resources, to replace virtual machinesdisabled by physical hardware problems and failures, and to ensure thatmultiple virtual machines supporting a high-availability virtualappliance are executing on multiple physical computer systems so thatthe services provided by the virtual appliance are continuouslyaccessible, even when one of the multiple virtual appliances becomescompute bound, data-access bound, suspends execution, or fails. Thus,the virtual data center layer of abstraction provides avirtual-data-center abstraction of physical data centers to simplifyprovisioning, launching, and maintenance of virtual machines and virtualappliances as well as to provide high-level, distributed functionalitiesthat involve pooling the resources of individual physical servers andmigrating virtual machines among physical servers to achieve loadbalancing, fault tolerance, and high availability.

FIG. 8 illustrates virtual-machine components of a virtual-data-centermanagement server and physical servers of a physical data center abovewhich a virtual-data-center interface is provided by thevirtual-data-center management server. The virtual-data-centermanagement server 802 and a virtual-data-center database 804 comprisethe physical components of the management component of the virtual datacenter. The virtual-data-center management server 802 includes ahardware layer 806 and virtualization layer 808, and runs avirtual-data-center management-server virtual machine 810 above thevirtualization layer. Although shown as a single server in FIG. 8, thevirtual-data-center management server (“VDC management server”) mayinclude two or more physical server computers that support multipleVDC-management-server virtual appliances. The virtual machine 810includes a management-interface component 812, distributed services 814,core services 816, and a host-management interface 818. The managementinterface is accessed from any of various computers, such as the PC 708shown in FIG. 7. The management interface allows the virtual-data-centeradministrator to configure a virtual data center, provision virtualmachines, collect statistics and view log files for the virtual datacenter, and to carry out other, similar management tasks. Thehost-management interface 818 interfaces to virtual-data-center agents824, 825, and 826 that execute as virtual machines within each of thephysical servers of the physical data center that is abstracted to avirtual data center by the VDC management server.

The distributed services 814 include a distributed-resource schedulerthat assigns virtual machines to execute within particular physicalservers and that migrates virtual machines in order to most effectivelymake use of computational bandwidths, data-storage capacities, andnetwork capacities of the physical data center. The distributed servicesfurther include a high-availability service that replicates and migratesvirtual machines in order to ensure that virtual machines continue toexecute despite problems and failures experienced by physical hardwarecomponents. The distributed services also include a live-virtual-machinemigration service that temporarily halts execution of a virtual machine,encapsulates the virtual machine in an OVF package, transmits the OVFpackage to a different physical server, and restarts the virtual machineon the different physical server from a virtual-machine state recordedwhen execution of the virtual machine was halted. The distributedservices also include a distributed backup service that providescentralized virtual-machine backup and restore.

The core services provided by the VDC management server include hostconfiguration, virtual-machine configuration, virtual-machineprovisioning, generation of virtual-data-center alarms and events,ongoing event logging and statistics collection, a task scheduler, and aresource-management module. Each physical server 820-822 also includes ahost-agent virtual machine 828-830 through which the virtualizationlayer can be accessed via a virtual-infrastructure applicationprogramming interface (“API”). This interface allows a remoteadministrator or user to manage an individual server through theinfrastructure API. The virtual-data-center agents 824-826 accessvirtualization-layer server information through the host agents. Thevirtual-data-center agents are primarily responsible for offloadingcertain of the virtual-data-center management-server functions specificto a particular physical server to that physical server. Thevirtual-data-center agents relay and enforce resource allocations madeby the VDC management server, relay virtual-machine provisioning andconfiguration-change commands to host agents, monitor and collectperformance statistics, alarms, and events communicated to thevirtual-data-center agents by the local host agents through theinterface API, and to carry out other, similar virtual-data-managementtasks.

The virtual-data-center abstraction provides a convenient and efficientlevel of abstraction for exposing the computational resources of acloud-computing facility to cloud-computing-infrastructure users. Acloud-director management server exposes virtual resources of acloud-computing facility to cloud-computing-infrastructure users. Inaddition, the cloud director introduces a multi-tenancy layer ofabstraction, which partitions VDCs into tenant-associated VDCs that caneach be allocated to a particular individual tenant or tenantorganization, both referred to as a “tenant.” A given tenant can beprovided one or more tenant-associated VDCs by a cloud director managingthe multi-tenancy layer of abstraction within a cloud-computingfacility. The cloud services interface (308 in FIG. 3) exposes avirtual-data-center management interface that abstracts the physicaldata center.

FIG. 9 illustrates a cloud-director level of abstraction. In FIG. 9,three different physical data centers 902-904 are shown below planesrepresenting the cloud-director layer of abstraction 906-908. Above theplanes representing the cloud-director level of abstraction,multi-tenant virtual data centers 910-912 are shown. The resources ofthese multi-tenant virtual data centers are securely partitioned inorder to provide secure virtual data centers to multiple tenants, orcloud-services-accessing organizations. For example, acloud-services-provider virtual data center 910 is partitioned into fourdifferent tenant-associated virtual-data centers within a multi-tenantvirtual data center for four different tenants 916-919. Eachmulti-tenant virtual data center is managed by a cloud directorcomprising one or more cloud-director servers 920-922 and associatedcloud-director databases 924-926. Each cloud-director server or serversruns a cloud-director virtual appliance 930 that includes acloud-director management interface 932, a set of cloud-directorservices 934, and a virtual-data-center management-server interface 936.The cloud-director services include an interface and tools forprovisioning multi-tenant virtual data center virtual data centers onbehalf of tenants, tools and interfaces for configuring and managingtenant organizations, tools and services for organization of virtualdata centers and tenant-associated virtual data centers within themulti-tenant virtual data center, services associated with template andmedia catalogs, and provisioning of virtualization networks from anetwork pool. Templates are virtual machines that each contains an OSand/or one or more virtual machines containing applications. A templatemay include much of the detailed contents of virtual machines andvirtual appliances that are encoded within OVF packages, so that thetask of configuring a virtual machine or virtual appliance issignificantly simplified, requiring only deployment of one OVF package.These templates are stored in catalogs within a tenant's virtual-datacenter. These catalogs are used for developing and staging new virtualappliances and published catalogs are used for sharing templates invirtual appliances across organizations. Catalogs may include OS imagesand other information relevant to construction, distribution, andprovisioning of virtual appliances.

Considering FIGS. 7 and 9, the VDC-server and cloud-director layers ofabstraction can be seen, as discussed above, to facilitate employment ofthe virtual-data-center concept within private and public clouds.However, this level of abstraction does not fully facilitate aggregationof single-tenant and multi-tenant virtual data centers intoheterogeneous or homogeneous aggregations of cloud-computing facilities.The current document is directed to providing an additional layer ofabstraction to facilitate aggregation of cloud-computing facilities.

FIG. 10 illustrates virtual-cloud-connector nodes (“VCC nodes”) and aVCC server, components of a distributed system that provides multi-cloudaggregation and that includes a cloud-connector server andcloud-connector nodes that cooperate to provide services that aredistributed across multiple clouds. In FIG. 10, seven differentcloud-computing facilities are illustrated 1002-1008. Cloud-computingfacility 1002 is a private multi-tenant cloud with a cloud director 1010that interfaces to a VDC management server 1012 to provide amulti-tenant private cloud comprising multiple tenant-associated virtualdata centers. The remaining cloud-computing facilities 1003-1008 may beeither public or private cloud-computing facilities and may besingle-tenant virtual data centers, such as virtual data centers 1003and 1006, multi-tenant virtual data centers, such as multi-tenantvirtual data centers 1004 and 1007-1008, or any of various differentkinds of third-party cloud-services facilities, such as third-partycloud-services facility 1005. An additional component, the VCC server1014, acting as a controller is included in the private cloud-computingfacility 1002 and interfaces to a VCC node 1016 that runs as a virtualappliance within the cloud director 1010. A VCC server may also run as avirtual appliance within a VDC management server that manages asingle-tenant private cloud. The VCC server 1014 additionallyinterfaces, through the Internet, to VCC node virtual appliancesexecuting within remote VDC management servers, remote cloud directors,or within the third-party cloud services 1018-1023. The VCC serverprovides a VCC server interface that can be displayed on a local orremote terminal, PC, or other computer system 1026 to allow acloud-aggregation administrator or other user to accessVCC-server-provided aggregate-cloud distributed services. In general,the cloud-computing facilities that together form amultiple-cloud-computing aggregation through distributed servicesprovided by the VCC server and VCC nodes are geographically andoperationally distinct.

FIG. 11 illustrates the VCC server and VCC nodes in a slightly differentfashion than the VCC server and VCC nodes are illustrated in FIG. 10. InFIG. 11, the VCC server virtual machine 1102 is shown executing within aVCC server 1104, one or more physical servers located within a privatecloud-computing facility. The VCC-server virtual machine includes aVCC-server interface 1106 through which a terminal, PC, or othercomputing device 1108 interfaces to the VCC server. The VCC server, uponrequest, displays a VCC-server user interface on the computing device1108 to allow a cloud-aggregate administrator or other user to accessVCC-server-provided functionality. The VCC-server virtual machineadditionally includes a VCC-node interface 1108 through which the VCCserver interfaces to VCC-node virtual appliances that execute within VDCmanagement servers, cloud directors, and third-party cloud-computingfacilities. As shown in FIG. 11, in one implementation, a VCC-nodevirtual machine is associated with each organization configured withinand supported by a cloud director. Thus, VCC nodes 1112-1114 execute asvirtual appliances within cloud director 1116 in association withorganizations 1118-1120, respectively. FIG. 11 shows a VCC-node virtualmachine 1122 executing within a third-party cloud-computing facility anda VCC-node virtual machine 1124 executing within a VDC managementserver. The VCC server, including the services provided by theVCC-server virtual machine 1102, in conjunction with the VCC-nodevirtual machines running within remote VDC management servers, clouddirectors, and within third-party cloud-computing facilities, togetherprovide functionality distributed among the cloud-computing-facilitycomponents of either heterogeneous or homogeneous cloud-computingaggregates.

FIG. 12 illustrates one implementation of a VCC node. The VCC node 1200is a web service that executes within an Apache/Tomcat container thatruns as a virtual appliance within a cloud director, VDC managementserver, or third-party cloud-computing server. The VCC node exposes webservices 1202 to a remote VCC server via REST APIs accessed through therepresentational state transfer (“REST”) protocol 1204 via a hypertexttransfer protocol (“HTTP”) proxy server 1206. The REST protocol usesHTTP requests to post data and requests for services, read data andreceive service-generated responses, and delete data. The web services1202 comprise a set of internal functions that are called to execute theREST APIs 1204. Authorization services are provided by a spring securitylayer 1208. The internal functions that implement the web servicesexposed by the REST APIs employ a metadata/object-store layerimplemented using an SQL Server database 1210-1212, a storage layer 1214with adapters 1216-1219 provides access to data stores 1220, filesystems 1222, the virtual-data-center management-server managementinterface 1224, and the cloud-director management interface 1226. Theseadapters may additional include adapters to 3rd-party cloud managementservices, interfaces, and systems. The internal functions that implementthe web services may also access a message protocol 1230 and networktransfer services 1232 that allow for transfer of OVF packages and otherfiles securely between VCC nodes via virtual networks 1234 thatvirtualize underlying physical networks 1236. The message protocol 1230and network transfer services 1232 together provide for secure datatransfer, multipart messaging, and checkpoint-restart data transfer thatallows failed data transfers to be restarted from most recentcheckpoints, rather than having to be entirely retransmitted.

FIG. 13 illustrates electronic communications between a client andserver computer. The following discussion of FIG. 13 provides anoverview of electronic communications. This is, however, a very largeand complex subject area, a full discussion of which would likely runfor many hundreds or thousands of pages. The following overview isprovided as a basis for discussing the REST architecture, with referenceto subsequent figures. In FIG. 13, a client computer 1302 is shown to beinterconnected with a server computer 1304 via local communication links1306 and 1308 and a complex distributed intermediary communicationssystem 1310, such as the Internet. This complex communications systemmay include a large number of individual computer systems and many typesof electronic communications media, including wide-area networks, publicswitched telephone networks, wireless communications, satellitecommunications, and many other types of electronics-communicationssystems and intermediate computer systems, routers, bridges, and otherdevice and system components. Both the server and client computers areshown to include three basic internal layers including an applicationslayer 1312 in the client computer and a corresponding applications andservices layer 1314 in the server computer, an operating-system layer1316 and 1318, and a hardware layer 1320 and 1322. The server computer1304 is additionally associated with an internal, peripheral, or remotedata-storage subsystem 1324. The hardware layers 1320 and 1322 mayinclude the components discussed above with reference to FIG. 1 as wellas many additional hardware components and subsystems, such as powersupplies, cooling fans, switches, auxiliary processors, and many othermechanical, electrical, electromechanical, andelectro-optical-mechanical components. The operating system 1316 and1318 represents the general control system of both a client computer1302 and a server computer 1304. The operating system interfaces to thehardware layer through a set of registers that, under processor control,are used for transferring data, including commands and storedinformation, between the operating system and various hardwarecomponents. The operating system also provides a complex executionenvironment in which various application programs, including databasemanagement systems, web browsers, web services, and other applicationprograms execute. In many cases, modern computer systems employ anadditional layer between the operating system and the hardware layer,referred to as a “virtualization layer,” that interacts directly withthe hardware and provides a virtual-hardware-execution environment forone or more operating systems.

Client systems may include any of many types of processor-controlleddevices, including tablet computers, laptop computers, mobile smartphones, and other such processor-controlled devices. These various typesof clients may include only a subset of the components included in adesktop personal component as well components not generally included indesktop personal computers.

Electronic communications between computer systems generally comprisespackets of information, referred to as datagrams, transferred fromclient computers to server computers and from server computers to clientcomputers. In many cases, the communications between computer systems iscommonly viewed from the relatively high level of an application programwhich uses an application-layer protocol for information transfer.However, the application-layer protocol is implemented on top ofadditional layers, including a transport layer, Internet layer, and linklayer. These layers are commonly implemented at different levels withincomputer systems. Each layer is associated with a protocol for datatransfer between corresponding layers of computer systems. These layersof protocols are commonly referred to as a “protocol stack.” In FIG. 13,a representation of a common protocol stack 1330 is shown below theinterconnected server and client computers 1304 and 1302. The layers areassociated with layer numbers, such as layer number “1” 1332 associatedwith the application layer 1334. These same layer numbers are used inthe depiction of the interconnection of the client computer 1302 withthe server computer 1304, such as layer number “1” 1332 associated witha horizontal dashed line 1336 that represents interconnection of theapplication layer 1312 of the client computer with theapplications/services layer 1314 of the server computer through anapplication-layer protocol. A dashed line 1336 representsinterconnection via the application-layer protocol in FIG. 13, becausethis interconnection is logical, rather than physical. Dashed-line 1338represents the logical interconnection of the operating-system layers ofthe client and server computers via a transport layer. Dashed line 1340represents the logical interconnection of the operating systems of thetwo computer systems via an Internet-layer protocol. Finally, links 1306and 1308 and cloud 1310 together represent the physical communicationsmedia and components that physically transfer data from the clientcomputer to the server computer and from the server computer to theclient computer. These physical communications components and mediatransfer data according to a link-layer protocol. In FIG. 13, a secondtable 1342 aligned with the table 1330 that illustrates the protocolstack includes example protocols that may be used for each of thedifferent protocol layers. The hypertext transfer protocol (“HTTP”) maybe used as the application-layer protocol 1344, the transmission controlprotocol (“TCP”) 1346 may be used as the transport-layer protocol, theInternet protocol 1348 (“IP”) may be used as the Internet-layerprotocol, and, in the case of a computer system interconnected through alocal Ethernet to the Internet, the Ethernet/IEEE 802.3u protocol 1350may be used for transmitting and receiving information from the computersystem to the complex communications components of the Internet. Withincloud 1310, which represents the Internet, many additional types ofprotocols may be used for transferring the data between the clientcomputer and server computer.

Consider the sending of a message, via the HTTP protocol, from theclient computer to the server computer. An application program generallymakes a system call to the operating system and includes, in the systemcall, an indication of the recipient to whom the data is to be sent aswell as a reference to a buffer that contains the data. The data andother information are packaged together into one or more HTTP datagrams,such as datagram 1352. The datagram may generally include a header 1354as well as the data 1356, encoded as a sequence of bytes within a blockof memory. The header 1354 is generally a record composed of multiplebyte-encoded fields. The call by the application program to anapplication-layer system call is represented in FIG. 13 by solidvertical arrow 1358. The operating system employs a transport-layerprotocol, such as TCP, to transfer one or more application-layerdatagrams that together represent an application-layer message. Ingeneral, when the application-layer message exceeds some thresholdnumber of bytes, the message is sent as two or more transport-layermessages. Each of the transport-layer messages 1360 includes atransport-layer-message header 1362 and an application-layer datagram1352. The transport-layer header includes, among other things, sequencenumbers that allow a series of application-layer datagrams to bereassembled into a single application-layer message. The transport-layerprotocol is responsible for end-to-end message transfer independent ofthe underlying network and other communications subsystems, and isadditionally concerned with error control, segmentation, as discussedabove, flow control, congestion control, application addressing, andother aspects of reliable end-to-end message transfer. Thetransport-layer datagrams are then forwarded to the Internet layer viasystem calls within the operating system and are embedded withinInternet-layer datagrams 1364, each including an Internet-layer header1366 and a transport-layer datagram. The Internet layer of the protocolstack is concerned with sending datagrams across the potentially manydifferent communications media and subsystems that together comprise theInternet. This involves routing of messages through the complexcommunications systems to the intended destination. The Internet layeris concerned with assigning unique addresses, known as “IP addresses,”to both the sending computer and the destination computer for a messageand routing the message through the Internet to the destinationcomputer. Internet-layer datagrams are finally transferred, by theoperating system, to communications hardware, such as anetwork-interface controller (“NIC”) which embeds the Internet-layerdatagram 1364 into a link-layer datagram 1370 that includes a link-layerheader 1372 and generally includes a number of additional bytes 1374appended to the end of the Internet-layer datagram. The link-layerheader includes collision-control and error-control information as wellas local-network addresses. The link-layer packet or datagram 1370 is asequence of bytes that includes information introduced by each of thelayers of the protocol stack as well as the actual data that istransferred from the source computer to the destination computeraccording to the application-layer protocol.

RESTful APIs

Next, the RESTful approach to web-service APIs is described, beginningwith FIG. 14. FIG. 14 illustrates the role of resources in RESTful APIs.In FIG. 14, and in subsequent figures, a remote client 1402 is shown tobe interconnected and communicating with a service provided by one ormore service computers 1404 via the HTTP protocol 1406. Many RESTfulAPIs are based on the HTTP protocol. Thus, the focus is on theapplication layer in the following discussion. However, as discussedabove with reference to FIG. 13, the remote client 1402 and serviceprovided by one or more server computers 1404 are, in fact, physicalsystems with application, operating-system, and hardware layers that areinterconnected with various types of communications media andcommunications subsystems, with the HTTP protocol the highest-levellayer in a protocol stack implemented in the application,operating-system, and hardware layers of client computers and servercomputers. The service may be provided by one or more server computers,as discussed above in a preceding section. As one example, a number ofservers may be hierarchically organized as various levels ofintermediary servers and end-point servers. However, the entirecollection of servers that together provide a service are addressed by adomain name included in a uniform resource identifier (“URI”), asfurther discussed below. A RESTful API is based on a small set of verbs,or operations, provided by the HTTP protocol and on resources, eachuniquely identified by a corresponding URL Resources are logicalentities, information about which is stored on one or more servers thattogether comprise a domain. URIs are the unique names for resources. Aresource about which information is stored on a server that is connectedto the Internet has a unique URI that allows that information to beaccessed by any client computer also connected to the Internet withproper authorization and privileges. URIs are thus globally uniqueidentifiers, and can be used to specify resources on server computersthroughout the world. A resource may be any logical entity, includingpeople, digitally encoded documents, organizations, and other suchentities that can be described and characterized by digitally encodedinformation. A resource is thus a logical entity. Digitally encodedinformation that describes the resource and that can be accessed by aclient computer from a server computer is referred to as a“representation” of the corresponding resource. As one example, when aresource is a web page, the representation of the resource may be ahypertext markup language (“HTML”) encoding of the resource. As anotherexample, when the resource is an employee of a company, therepresentation of the resource may be one or more records, eachcontaining one or more fields, that store information characterizing theemployee, such as the employee's name, address, phone number, job title,employment history, and other such information.

In the example shown in FIG. 14, the web servers 1404 provides a RESTfulAPI based on the HTTP protocol 1406 and a hierarchically organized setof resources 1408 that allow clients of the service to accessinformation about the customers and orders placed by customers of theAcme Company. This service may be provided by the Acme Company itself orby a third-party information provider. All of the customer and orderinformation is collectively represented by a customer informationresource 1410 associated with the URI “http://www.acme.com/customerInfo”1412. As discussed further, below, this single URI and the HTTP protocoltogether provide sufficient information for a remote client computer toaccess any of the particular types of customer and order informationstored and distributed by the service 1404. A customer informationresource 1410 represents a large number of subordinate resources. Thesesubordinate resources include, for each of the customers of the AcmeCompany, a customer resource, such as customer resource 1414. All of thecustomer resources 1414-1418 are collectively named or specified by thesingle URI “http://www.acme.com/customerInfo/customers” 1420. Individualcustomer resources, such as customer resource 1414, are associated withcustomer-identifier numbers and are each separately addressable bycustomer-resource-specific URIs, such as URI“http://www.acine.com/customerInfo/customers/361” 1422 which includesthe customer identifier “361” for the customer represented by customerresource 1414. Each customer may be logically associated with one ormore orders. For example, the customer represented by customer resource1414 is associated with three different orders 1424-1426, eachrepresented by an order resource. All of the orders are collectivelyspecified or named by a single URI“http://www.acme.com/customerInfo/orders” 1436. All of the ordersassociated with the customer represented by resource 1414, ordersrepresented by order resources 1424-1426, can be collectively specifiedby the URI “http://www.acme.com/customerInfo/customers/361/orders” 1438.A particular order, such as the order represented by order resource1424, may be specified by a unique URI associated with that order, suchas URI “http://www.acme.com/customerInfo/customers/361/orders/1” 1440,where the final “1” is an order number that specifies a particular orderwithin the set of orders corresponding to the particular customeridentified by the customer identifier “361.”

In one sense, the URIs bear similarity to path names to files in filedirectories provided by computer operating systems. However, it shouldbe appreciated that resources, unlike files, are logical entities ratherthan physical entities, such as the set of stored bytes that togethercompose a file within a computer system. When a file is accessed througha path name, a copy of a sequence of bytes that are stored in a memoryor mass-storage device as a portion of that file are transferred to anaccessing entity. By contrast, when a resource is accessed through aURI, a server computer returns a digitally encoded representation of theresource, rather than a copy of the resource. For example, when theresource is a human being, the service accessed via a URI specifying thehuman being may return alphanumeric encodings of various characteristicsof the human being, a digitally encoded photograph or photographs, andother such information. Unlike the case of a file accessed through apath name, the representation of a resource is not a copy of theresource, but is instead some type of digitally encoded information withrespect to the resource.

In the example RESTful API illustrated in FIG. 14, a client computer canuse the verbs, or operations, of the HTTP protocol and the top-level URI1412 to navigate the entire hierarchy of resources 1408 in order toobtain information about particular customers and about the orders thathave been placed by particular customers.

FIGS. 15A-D illustrate four basic verbs, or operations, provided by theHTTP application-layer protocol used in RESTful applications. RESTfulapplications are client/server protocols in which a client issues anHTTP request message to a service or server and the service or serverresponds by returning a corresponding HTTP response message. FIGS. 15A-Duse the illustration conventions discussed above with reference to FIG.14 with regard to the client, service, and HTTP protocol. For simplicityand clarity of illustration, in each of these figures, a top portionillustrates the request and a lower portion illustrates the response.The remote client 1502 and service 1504 are shown as labeled rectangles,as in FIG. 14. A right-pointing solid arrow 1506 represents sending ofan HTTP request message from a remote client to the service and aleft-pointing solid arrow 1508 represents sending of a response messagecorresponding to the request message by the service to the remoteclient. For clarity and simplicity of illustration, the service 1504 isshown associated with a few resources 1510-1512.

FIG. 15A illustrates the GET request and a typical response. The GETrequest requests the representation of a resource identified by a URIfrom a service. In the example shown in FIG. 15A, the resource 1510 isuniquely identified by the URI “http://www.acme.com/item1” 1516. Theinitial substring “http://www.acme.com” is a domain name that identifiesthe service. Thus, URI 1516 can be thought of as specifying the resource“item1” that is located within and managed by the domain “www.acme.com.”The GET request 1520 includes the command “GET” 1522, a relativeresource identifier 1524 that, when appended to the domain name,generates the URI that uniquely identifies the resource, and in anindication of the particular underlying application-layer protocol 1526.A request message may include one or more headers, or key/value pairs,such as the host header 1528 “Host:www.acme.com” that indicates thedomain to which the request is directed. There are many differentheaders that may be included. In addition, a request message may alsoinclude a request-message body. The body may be encoded in any ofvarious different self-describing encoding languages, often JSON, XML,or HTML. In the current example, there is no request-message body. Theservice receives the request message containing the GET command,processes the message, and returns a corresponding response message1530. The response message includes an indication of theapplication-layer protocol 1532, a numeric status 1534, a texturalstatus 1536, various headers 1538 and 1540, and, in the current example,a body 1542 that includes the HTML encoding of a web page. Again,however, the body may contain any of many different types ofinformation, such as a JSON object that encodes a personnel file,customer description, or order description. GET is the most fundamentaland generally most often used verb, or function, of the HTTP protocol.

FIG. 15B illustrates the POST HTTP verb. In FIG. 15B, the client sends aPOST request 1546 to the service that is associated with the URI“http://www.acme.com/item1.” In many RESTful APIs, a POST requestmessage requests that the service create a new resource subordinate tothe URI associated with the POST request and provide a name andcorresponding URI for the newly created resource. Thus, as shown in FIG.15B, the service creates a new resource 1548 subordinate to resource1510 specified by URI “http://www.acme.com/item1,” and assigns anidentifier “36” to this new resource, creating for the new resource theunique URI “http://www.acme.com/item1/36” 1550. The service thentransmits a response message 1552 corresponding to the POST request backto the remote client. In addition to the application-layer protocol,status, and headers 1554, the response message includes a locationheader 1556 with the URI of the newly created resource. According to theHTTP protocol, the POST verb may also be used to update existingresources by including a body with update information. However, RESTfulAPIs generally use POST for creation of new resources when the names forthe new resources are determined by the service. The POST request 1546may include a body containing a representation or partial representationof the resource that may be incorporated into stored information for theresource by the service.

FIG. 15C illustrates the PUT HTTP verb. In RESTful APIs, the PUT HTTPverb is generally used for updating existing resources or for creatingnew resources when the name for the new resources is determined by theclient, rather than the service. In the example shown in FIG. 15C, theremote client issues a PUT HTTP request 1560 with respect to the URI“http://www.acme.com/item1/36” that names the newly created resource1548. The PUT request message includes a body with a JSON encoding of arepresentation or partial representation of the resource 1562. Inresponse to receiving this request, the service updates resource 1548 toinclude the information 1562 transmitted in the PUT request and thenreturns a response corresponding to the PUT request 1564 to the remoteclient.

FIG. 15D illustrates the DELETE HTTP verb. In the example shown in FIG.15D, the remote client transmits a DELETE HTTP request 1570 with respectto URI “http://www.acme.com/item1/36” that uniquely specifies newlycreated resource 1548 to the service. In response, the service deletesthe resource associated with the URL and returns a response message1572.

As further discussed below, and as mentioned above, a service mayreturn, in response messages, various different links, or URIs, inaddition to a resource representation. These links may indicate, to theclient, additional resources related in various different ways to theresource specified by the URI associated with the corresponding requestmessage. As one example, when the information returned to a client inresponse to a request is too large for a single HTTP response message,it may be divided into pages, with the first page returned along withadditional links, or URIs, that allow the client to retrieve theremaining pages using additional GET requests. As another example, inresponse to an initial GET request for the customer info resource (1410in FIG. 14), the service may provide URIs 1420 and 1436 in addition to arequested representation to the client, using which the client may beginto traverse the hierarchical resource organization in subsequent GETrequests.

Returning of hyperlinks, or additional URIs, by a service to a clientcan also be used to dynamically extend an initial API. FIGS. 16A-Billustrate extension of the simple RESTful API discussed above withreference to FIGS. 14 and 15A-D. FIG. 16A illustrates the hierarchicalorganization of resources in the initial RESTful API 502. FIG. 16Billustrates an extended RESTful API 1604. The extension involves addingan additional set of resources, at the lowest hierarchical level 1606,that represent the status of orders. Thus, each order resource, such asorder resource 1608, is now associated with a corresponding statusresource 1610. This RESTful API extension may be undertaken by theservice or may be undertaken by an appropriately authorized client.However, in either case, once the extension is carried out, the serviceneeds to inform clients unaware that the extension has occurred of theavailability of additional information. This can be accomplished byincluding additional links in responses.

FIG. 17 illustrates the use of additional links and responses in orderto inform a client of newly available resources as a result of extensionof a RESTful API. A client unaware of the extension may requestinformation about an order using a GET request 1702. In response, theservice transmits the response message 1704 back to the client. The bodyof the response message 1706 is encoded in JSON. It includes a dataobject with numerous key/value pairs that describe or provide arepresentation for the order specified by the URI 1708 contained in theGET request. These key/value pairs 1710 provide a product code, productdescription, and other such information. In addition, the data objectincludes a links object 1712 that includes several links 1714 and 1716.The first link 1714 is a self link, referring back to the original URI1708 included in the GET request. The second link 1716 is a related linkto the self link and is the URI for the new status resource associatedwith the order specified by the URI 1708. Using this link, the clientcan then submit a second GET request 1717 to the service in order toobtain the status information for the order, which is returned in asecond response message 1718 by the service. Thus, the use of additionallinks, or hyperlinks, in the body of responses to GET requests caninform clients of RESTful API extensions. A server can add any of manydifferent types of additional links to facilitate extension of APIs andto facilitate navigation, via subsequent GET requests, carried out byclients as they traverse the hierarchically organized resourcesavailable through the API.

FIGS. 18A-C illustrate the type of functionality present within a serverthat handles requests, including GET requests from clients. FIG. 18Aillustrates a listener process within a server that receives HTTPrequests through a communications port provided by a protocol stack,such as the protocol stack 1330 shown in FIG. 13. A server may employnumerous listener processes to listen for HTTP requests through one ormore ports and dispatch the requests for servicing. In FIG. 18A, theprocess listens to a communications port for a new HTTP request, in step1802 and, when a request becomes available, processes the request instep 1804 by calling a routine “process request.” The processing may beundertaken by the listener process or by another process notified by thelistener process to dequeue the next request from a memory queue.

FIG. 18B illustrates the “process request” routine called in step 1804in FIG. 18A. The request is received, either as a reference or as aresult of dequeuing the process from a memory queue, by the routine“processRequest” in step 1806. Then, in a series of conditional steps,including steps 1808-1811, the routine determines, in a sequence ofconditional steps, whether or not the type of request is one of a numberof different HTTP request types and, upon determining the request typeis equal to a particular request type, calls an appropriate routine tohandle the request. For example, when the request type is GET, asdetermined in step 1808, then the routine “processRequest” calls a “GET”routine, in step 1812, to process the GET request. The routine“processRequest” may additionally include a default handler, shown asstep 1814 in FIG. 18B, when the routine fails to determine the type ofrequest or encounters any of various errors.

FIG. 18C illustrates a partial implementation of the “GET” routinecalled in step 1812 of FIG. 18B. In step 1820, the routine “GET”receives and parses a GET request provided to the routine via areference or copy argument. In step 1822, the routine “GET” determineswhether or not the requested resource, specified by the URI included inthe GET request, is available. When, as determined in step 1824, theresource is not available, the “GET” routine, in step 1826, prepares andreturns an error response to the requesting client. Otherwise, in step1828, the routine “GET” generates a representation of the requestedresource within a response message corresponding to the GET request.Ellipses 1830 indicate that additional steps may be executed in order toprocess a GET request and complete the corresponding response message.In the case that the initial API has been extended, as discussed abovewith reference to FIGS. 16A-B and 17, the “GET” routine may undertakethe addition of links to the response message. As shown in FIG. 18C, asan example, the “GET” routine includes additional conditional steps,such as conditional step 1832, in which the “GET” routine determineswhether or not the URI of the requested resource is equal to aparticular value, with the particular values indicated by symbols “X”and “Y” in FIG. 18C, and, when the URI is equal to the particular URI,the “GET” routine then adds one or more links to the response messageappropriate for a GET request that has requested the representation ofthe particular resource. For example, in FIG. 18C, when the URI for therequested resource equals a value X, as determined in step 1832, the“GET” routine calls an additional routine in step 1834 to add one ormore links to the response message appropriate for a GET requestdirected to the URI. Thus, in the example of FIGS. 16A-B, a GET requestfor a particular order may result in a call to a routine to add a linkto the status resource associated with the order to the responsemessage. Finally, in step 1836, a completed response message is returnedby the routine “GET.”

As can be appreciated from the high-level control-flow diagram 18A-C, inthe approach illustrated in these figures, the server bears a rathersignificant processing burden with respect to the extension of an API.Extension of the API based on the resources shown in FIG. 16A, forexample, to the resources shown in FIG. 16B may require significantspecialized control logic, such as a series of conditionals and routinecalls similar to step 1832 and 1834 in FIG. 18C, in order to providelinks to the new status resources in response messages corresponding toGET requests for orders. Should the server wish to add additional linksto GET requests for the collection of orders, then even greatercomputational and development burdens would ensue. Although it may bepossible to write somewhat more generic logic for handling the simpleextension illustrated in FIGS. 16A-B, the somewhat more generic logicwould still require significant development efforts. For a more complexAPI featuring significantly more types of resources and more resourcesof particular types, the computational and development burdens maybecome significant and even prohibitive. Were clients to extend the API,through PUT requests, then an even more complex distributed effort wouldbe needed to enable clients to alter server logic for processingrequests and adding additional related links to request responses.

The a new approach for extending RESTful APIs without incurringsignificant computational and development overheads, as discussed abovewith reference to FIG. 18C, is next discussed. The approach uses a linkregistry. The link registry is a collection of link-registry entries,each of which represents additional links added to response messagescorresponding to request messages directed to a particular resource.FIGS. 19A-B illustrate an example link registry. FIG. 19A illustratesone implementation of a link-registry entry. The link-registry entry1902 is a variable-length record that includes numerous fields. Inexample link-registry entry 1902, these fields include: (1) ResourceURI1904, a field containing a symbol-string representation of the URI towhich a request message may be directed; (2) numKvp 1906, the number ofkey/value pairs that follow this field; (3) a number of key/value fieldpairs 1908-1909 equal to the value in the field numKvp 1906, the keyvalue pairs representing various types of conditions or constraints withrespect to returning additional links, further discussed below; (4)numLinks 1910, a field indicating the number of links that follow thisfield; and (5) a number of link fields, such as fields 1912 and 1914,equal to the value stored in the numLinks field 1910, each link fieldstoring a representation of a link to be added to the response message,such as the related link 1716 in FIG. 17. Note that each link mayinclude multiple key/value pairs, such as the three key/value pairsincluded in link 1716.

FIG. 19B illustrates the link registry. The link registry is a set oflinks 1916, with each cell in the column array 1916, such as cell 1918,representing a link-registry entry, such as link-registry entry 1902shown in FIG. 19A. The link registry may be implemented in manydifferent ways. For example, the link entries may be rows in arelational database table that is stored and managed by a relationaldatabase management system. Alternatively, the link-registry entries maybe stored in files associated with various types of indexes to allow forefficient location of particular link-registry entries. Thelink-registry entries may also be sorted with respect to the values ofone or more of the fields.

A link-registry entry may be represented in JSON. FIG. 20 illustrates aJSON encoding of a particular link-registry entry. Key/value pairs areused, in the JSON representation, to represent the various fields andvalues stored in the fields. For example, the Resource URI field (1904in FIG. 19) is represented by key “ResourceURI” 2002 and a particularsymbol-string representation of a particular URI 2004. Similarly, thenumKvp field (1906 in FIG. 19A) is represented by key 2006 and value2008.

In one implementation, the link registry is made accessible via a simplelink-registry API, which is a subset of the RESTful customer informationAPI discussed above with reference to FIGS. 14-16B. FIG. 21 showsexample method calls to a simple link-registry API. The link registry isassociated with a URI 2102. The link registry can be accessed using theGET, PUT, POST, and DELETE HTTP methods. Examples 2104 are provided inFIG. 21. The GET request 2106 can be used to obtain a representation ofthe entire link registry, including representations of all of thelink-registry entries. GET request 2108 may be used to find alink-registry entry corresponding to the collective customers resource.PUT request 2110 can be used to store or update the link-registry entryfor the customers collective resource, with the link-registry entryrepresented by symbol string “LinkRegistryEntry” 2112. The link-registryentry corresponding to the collective customers resource can be deletedusing the DELETE request 2114. The link-registry entry corresponding toa particular customer can be retrieved using GET request 2116.

The key/value pairs contained in a link-registry entry provides amechanism for directing additional links to response messages withparticular characteristics. For example, because resources are oftenhierarchically organized, it may be desirable, in certain cases, foradditional links to be added with respect to an intermediate-levelresource in the path of a lower-level resource. In other words, as shownin FIG. 20, when the value of the resource URI field is“http://www.acme.com/customerInfo/customers,” the creator of thelink-registry entry may desire for the additional link specified in thelink-registry entry to be added in a response message to a request for arepresentation of the resource“http://www.acme.com/customerInfo/customers/361/orders/1.” On the otherhand, the author of the link-registry entry may wish for the additionallinks only to be added when a representation of the specific resource“http://www.acme.com/customerInfo/customers” is requested in a requestmessage. In one scheme, illustrated in FIG. 20, the key “appliedTo” 2010may be associated with either the value “terminal” or “non-terminal.”When the value is “terminal,” then the specified additional links areadded only when the URI in the request message exactly matches the URIin the field “ResourceURI” of the link-registry entry. However, when thevalue is “non-terminal,” then a requested URI that contains the value ofthe “Resource URI” field as an initial substring should be responded towith a response message that contains the additional links. Otherkey/value pairs may indicate that the additional link should only beadded for response messages containing particular values and/or forrequest messages containing particular values. In the example shown inFIG. 20, the two key/value pairs 2012 specify that the additional link2014 should be included in a response message only when the body of theresponse message contains the key/value pairs city/Des Moines andstate/Iowa. In alternative implementations, a link-registry entry mayomit the numKvp and kvp keys and associated values and simply addadditional links into the response message corresponding to a requestmessage requesting a representation of a resource specified by a URIthat exactly matches the value of the “ResourceURI” field of thelink-registry entry. In yet alternative implementations, many differenttypes of additional filtering and selection specifiers may be includedin a link-registry entry, including range specifiers, relationaloperators, and other features common to, for example, relationaldatabase queries.

Authorization Service to Facilitate Multi-Tenancy in Cloud-CommutingFacilities

The VCC server and VCC nodes, discussed above, provide a virtualizationinfrastructure for the management of distributed, multi-tenant, virtualcloud-computing facilities, or virtual data centers. The managementfunctionality is referred to as the “cloud director.” The cloud directoris implemented as a distributed management infrastructure implemented bya combination of the VCC nodes and the VCC server and provides both amanagement interface as well as a suite of software services andfacilities accessible to cloud operators, system administrators, andother such clients of the cloud director.

While the cloud-director-provided functionalities and services are ofgreat benefit to cloud-director users, it is often the case that, forparticular virtual data centers and collections of virtual data centers,a cloud operator or system administrator may wish to supplement andenhance the facilities and services provided by the cloud director. Incertain cases, a cloud operator or system administrator may wish toprovide additional functionalities and services based on existingfunctionalities and services that were not developed for multi-tenantVDCs and which are unaware of, and cannot use, the multi-tenancyinfrastructure used by cloud-director-provided facilities and services.Currently, incorporating multi-tenancy-unaware facilities and servicesinto the cloud-director-provided functionalities and services can be arather daunting task for developers and users of the cloud-directorinterface.

FIGS. 22A-C illustrate a recently developed approach that allows forextension of the facilities and services provided by the cloud directorfor management of distributed, multi-tenant VDCs. As shown in FIG. 22A,the cloud director backend 2202 provides a collection of functionalitiesfor management and operational use of a collection of distributed,multi-tenant VDCs 2204. As discussed in preceding subsections, oneaspect of the cloud director is the provision of a user interface 2206through which cloud owners, system administrators, and othercloud-director users can access cloud-director functionalities andservices. In order to facilitate extensibility of the cloud director,the cloud director provides a single cloud application programminginterface (“API”) entry point 2208 that represents a RESTful interfaceto cloud-director functionalities and services. These functionalitiesand services can be broadly divided into a cloud-operator interface2210, provided through the cloud-API entrypoint, and a client interface2212, also provided through the cloud-API entrypoint 2208. Requests madethrough the cloud-API entrypoint 2208 are handled, in oneimplementation, by a cloud API server application 2214 that accesses alink registry 2216 that facilitates systematic extension of thecloud-API entrypoint, as discussed in the previous subsection.

As one example of the cloud-director-provided extensibility through thecloud-API entrypoint, a cloud operator may extend, through thecloud-operator interface 2210, the services provided by the clouddirector to include a newly developed or legacy backup service for VDCsthat can then be accessed, through the client interface 2212 portion ofthe cloud-API entrypoint, by various types of authorized users,including other cloud operators, system administrators, and other typesof privileged users. In addition, a cloud-director-provided service maybe redefined by extension.

FIG. 22B illustrates access, by a service client, to a serviceaccessible through the cloud-API entrypoint 2208 that has been added tothe cloud-director functionality by a cloud operator through thecloud-operator interface 2210. The client first accesses the cloud-APIentrypoint 2220 using a RESTful request that returns 2221, through aseries of responses and additional requests 2222 and 2224, a link to thenew backup service 2226 provided by a virtual server. This link allowsthe client to access 2228 the backup service. The cloud API serverapplication includes multiple filters 2230-2233, in one implementationimplemented as servlets, that perform initial processing of an HTTPrequest for the URIs within the cloud API entrypoint interface beforecalling a service extension. However, without some type of additionalfunctionality to incorporate the backup service into the multi-tenancyfunctionality of the cloud director, there would be no reasonable way tocontrol access to the new backup service. As one example, the cloudoperator who has extended the cloud director functionality to includethe new backup service might desire that the backup service only beaccessed by members of the cloud-operator's organization. In certaincases, the cloud operator may be willing to allow access to the backupservice by members of other organizations, but only to qualifiedsuperusers. The cloud-API entrypoint 2208 thus provides for extendingcloud-director functionality and services, but, by itself, does notprovide a means for access control needed in multi-tenant-VDCenvironments.

FIG. 22C illustrates an authorization service that allows formulti-tenancy-aware access control to services provided incloud-director-functionality extensions through the cloud-operatorinterface and cloud-API entrypoint. FIG. 22C continues from FIG. 22B.Having obtained a link to the new backup service, a client directs arequest 2228 to the new backup service 2226. The cloud API serverapplication 2214 directs the request through a series of filters(2230-2233 in FIG. 22C) 2236-2238. One of the filters authenticates therequests, preparing a security context that includes information relatedto the requestor. A subsequent filter, referred to as an extensionfilter, accesses 2240 an authorization service 2242 provided by thecloud-director backend 2202. The authorization service examinesinformation provided by the extension filter, including the securitycontext, requested URI, and HTTP request, to determine whether or notthe client is authorized to access the resource defined by the requestedURI. The authorization service returns 2246, to the extension filter, anindication of whether or not the client is authorized. When the clientis authorized, the extension filter passes 2248 the request to the newbackup service, which begins to process the request. As part of theinitial processing, the request is passed to a filter 2252, such as aservlet, which may access 2254 the authorization service to furtherauthorize the request. The authorization service examines informationprovided by the filter to determine whether or not the client isauthorized to access the service and returns 2256, to the filter, anindication of whether or not the client is authorized. When the clientis authorized, the filter passes the request to furtherservice-processing functionality 2260. Otherwise, the request is deniedfor lack of authorization. Ultimately, a response message is returned2262 to the client. Thus, the authorization service provides for accesscontrol to functionalities and services added to cloud-director-providedfunctionalities and services through the cloud-operator interface 2210and cloud-API entrypoint 2208.

FIG. 23 illustrates data objects maintained in an authorization-servicedatabase maintained by the cloud director to support the authorizationservice provided by the cloud-director backend to supportcloud-director-provided service extensions. The authorization-servicedatabase includes data objects that represent resource classes 2302,data objects that represent specific instances of given resource classcalled service resources 2304, data objects that represent particularactions or requests supported by resource classes 2306, organizations2308, users 2310, groups 2312, rights 2314, and roles 2316. There aretwo types of data objects that represent one of a set of primitive dataobjects: entities 2318, each data object of which can represent either aresource class or a service resource; and security principals 2320, eachdata object of which represents a user, group, right, or role. All ofthe above-mentioned types of data objects are used to support the typeof data object 2322 that represents what are referred to as “ACL rules.”

In FIG. 23, only a few fields for each data object are shown. The dataobjects may include additional fields or different fields in differentimplementations. A resource-class data object, such as data object 2330,includes the name of the resource class and a resource-class ID, orresource ID. Resource classes refer to general classes of services. Aparticular instance of a resource class is a service resource,represented by a service resource data object, such as data object 2332.A service resource data object includes a name, service ID, organizationID, and resource-class ID. As one example, backup services may bedescribed by a particular resource class represented by a resource-classdata object and each particular instance of the backup resource classmay each be described by a service resource data object. An entity dataobject, such as data object 2334, represents either a resource class ora service. An entity data object includes a name, entity ID, a typeindicating whether the entity represents a resource class or serviceresource, and the ID for the resource class or service resource. Anorganization data object, such as data object 2336, includes the name ofan organization and an organization ID associated with the organization.An action data object, such as data object 2338, includes the name of anaction, an action ID, the identifier of the entity that represents aresource class or service resource that supports the action, and a URIpattern that specifies one or more URIs that represent the action. Anaction is equivalent to a function provided by a type of resource classor by a specific service resource. For example, an action can be definedfor all backups or for a particular backup service resource. The URIpattern is a named regular expression, in which “(?<id>)” matches one ormore resource identifiers. The “id” tag may or may not be present in aparticular URI pattern. A user data object, such as data object 2340,includes the name of a user, the ID of the organization the user is amember of, and a user ID. A group data object, such as data object 2342,includes the name of the group, an organization ID for the organizationthat supports the group, and the group ID. A right data object, such asdata object 2344, includes the name of a right and an ID associated withthe right. A role data object, such as data object 2346, includes thename of a role and an ID associated with the role. A security-principlesdata object, such as data object 2348, includes a name, an ID associatedwith the security principle, a type indicating one of a user, group,right, or role, and the ID of the user, group, right, or role.

All of the above-discussed data objects, including resource-class,service resource, action, organization, user, group, right, role,entity, and security-principle data objects, are used to define ACLrules. An ACL rule represents a specific authorization to which arequest for a service can be matched in order to determine whether ornot the request is authorized. Each ACL rule, such as ACL rule 2350,includes an ID of an entity data object 2352, an ID of an action dataobject 2354, an ID of an organization data object 2356, the ID of thesecurity-principle data object 2358, and an ACL ID 2360. An ACL rulethus defines a relationship between a resource class or service, anaction, an organization, and one of a user, group, right, or role. Forexample, an ACL rule may authorize, for a specific action that may berequested of a specific service resource, access by a member of anorganization according to a particular security principle. The securityprinciple can specify a specific user of the organization that canaccess the action of the service, a group of users of the organizationthat can access the action of the service, a member of an organizationholding a particular right to access the action of the service, or amember of the organization that is associated with a particular role toaccess the action of the service. Various different types of rights androles can be defined. Roles may include job titles, job descriptions, orpositions, such as cloud owner or system administrator. Rights mayinclude various types of access rights, such as superuser-access rights.An ACL rule may specify authorization for a particular service resourceor for a particular resource class of which there are one or moreservice resource instances. The authorization-service database includesvarious types of default data objects, such as “any” data objects thatoperate much like wildcards in ACL rules and “shared” data objects. An“any” data object matches any other data object of the same type. A“shared” data object matches multiple organization data objects.

A subset of the cloud-operator interface (2210 in FIG. 22A) providedthrough the cloud-API entrypoint 2208 is an authorization-servicemanagement interface. This is a RESTful interface that allows for thecreation, retrieval, and deletion of various data objects in theauthorization-service database, described above with reference to FIG.23. FIG. 24 provides a table-based description of a portion of theauthorization-service management interface. For example, the table 2400,in a first column 2404, includes fourauthorization-service-management-interface (“ASMI”) operations 2402 thatallow for the creation of a resource class, retrieval of a descriptionof the resource class by resource-class ID, retrieval of a list ofresource classes, and deletion of a resource class. In a second column2406 of table 2400, the corresponding HTTP operations are shown for eachASMI operation. FIG. 25 shows, as one example, an HTTP/XMLimplementation of the ASMI operations 2402 in FIG. 24 associated withresource classes. Thus, the ASMI subset of the cloud-operator-interfacesubset of the cloud-API entrypoint allows authorized cloud-directorusers to create and delete primitive and collective data objects in theauthorization-service database as well as to create and delete ACLrules. Creation and deletion of data objects stored in theauthorization-service database alters operation of the authorizationservice and defines how requests to service actions are authorized bythe authorization service. Of course, access to the ASMI interface iscontrolled by the cloud-director multi-tenancy infrastructure so thatonly authorized and verified users can alter and manage portions of theauthorization-service interface.

FIG. 26 shows the call interface to the authorization service providedby the cloud director. This interface includes two authorization calls,including the call “authorize” 2602 and the call “authorize services”2604. The routine “authorize” 2602 returns a Boolean value indicatingwhether or not a particular request is authorized. This routine takes,as arguments, a definition of a service extension that provides externalfunctionality for the cloud, a security context that includesinformation with respect to the user or group making the request,including any rights or roles associated with the user or group, and theorganization to which the user or group belongs. The routine “authorize”also receives, as arguments, the requested URI and the correspondingHTTP method that implements the request, from which information aboutthe corresponding resource class and resource-class action can beobtained by matching the requested URI against the action's URI pattern.The routine “authorize services” 2604 carries out authorization of arequest with respect to multiple services. In this case, when therequest is authorized for all of the multiple services specified by aset of service definitions furnished as an argument to the routine, thenthe routine returns, but when the request fails authorization withrespect to one or more services, then the routine throws an unauthorizedexception to indicate that the request is not authorized. The routine“authorize services” is useful when a particular requested serviceresults in lower-level calls to other services, allowing the request tobe authorized with respect to all of the services invoked duringprocessing of the highest-level service.

FIGS. 27A-B provide control-flow diagrams that illustrate animplementation of the routine “authorize,” discussed above withreference to FIG. 26. In step 2702, the routine “authorize” receives aservice definition, security context, requested URI, and HTTP requestfurnished as arguments to the routine. In step 2704, the routine“authorize” matches the received service definition to a serviceresource data object in the authorization-service database and retrievesthe service resource ID from the data object. The matching may involvecomparing the name of the service resource provided in the servicedefinition to the name field of the service resource data object or mayinvolve more complex matching that involves different and/or additionalservice-data-object fields and different and/or additional portions ofthe service definition. In step 2705, the routine “authorize” determineswhether the service resource supports or provides an action that isrepresented by a regular expression, referred to as the “URI pattern,”that represents one or more URIs within the RESTful interface supportedby the cloud API entrypoint. When the service resource does not supportan action corresponding to the requested URI, a “false” Boolean value isreturned. Otherwise, in step 2706, the routine “authorize” obtains, fromthe security context received as an argument, the user ID and/or thegroup ID, role IDs, and right IDs associated with the request. Asdiscussed above, the security context is provided by the authenticationfiler to the extension filter within the cloud API server application.When the routine “authorize” fails to obtain a service resource ID andat least one of a user ID and group ID, as determined in step 2708, theroutine “authorize” returns “false” in step 2710. Otherwise, the routine“authorize 2” is called, in step 2718, to continue the authorizationprocess.

FIG. 27B provides a control-flow diagram of the routine “authorize 2,”called in step 2718 of FIG. 27A. In step 2720, the routine “authorize 2”processes the received requested URI and HTTP request in order toidentify the action to which the request that is being authorizedcorresponds by matching the action URI pattern to the request URI. Theroutine “authorize 2” then retrieves the corresponding resource actionID from the service-authorization database. In addition, when therequested URI includes an indication of the resource ID, the routine“authorize” extracts the resource ID from the URL The cloud director, incertain implementations, may maintain mappings between URIs and HTTPrequest types and resource-class actions. When the routine “authorize 2”fails to obtain an action ID for the request that is being authorized,as determined in step 2722, the routine “authorize 2” returns “false,”in step 2724, which becomes the return value for the routine“authorize,” called the routine “authorize 2” in step 2712. Otherwise,in the for-loop of steps 2726-2738, the routine “authorize 2” continuesto process ACL rules that include the action ID retrieved from theauthorization-service database until either an ACL rule is found thatauthorizes the request or no more ACL rules can be retrieved from theauthorization-service database. In the former case, the return value“true” is returned, in step 2733, which becomes the return value of theroutine “authorize.” In the latter case, the return value “false” isreturned in step 2730. For the currently considered ACL rule in thefor-loop of steps 2726-2738, the routine “authorize 2,” in step 2727,accesses the entity data object, organization data object, and securityprinciple data object, in the authorization-service database, thatcorrespond to the entity ID, organization ID, and security-principle IDin the ACL rule. When the entity data object is not an “ANY” data objectand the ID in the entity data object does not match a resource IDextracted from the requested URI or the service resource ID, asdetermined in step 2728, then control flows to step 2729, where theroutine “authorize 2” determines whether or not there are more ACL rulesto consider. When there are more ACL rules to consider, control flowsback to step 2728, for a next iteration of the for-loop of steps2726-2738. Otherwise, the value “false” is returned in step 2730. Whenthe entity data object is an “ANY” data object or the ID in the entitydata object matches a resource ID extracted from the requested URI orthe service resource ID, as determined in step 2728, the routine“authorize 2” determines, in step 2731, whether the organization dataobject is an “ANY” data object or whether the organization ID matchesthe organization ID obtained from the security context. When theorganization data object is not an “ANY” data object and theorganization ID does not match the organization ID obtained from thesecurity context, control flows to step 2729. Otherwise, the routine“authorize 2” determines, in step 2732, whether the security principledata object is an “ANY” data object. When the security principle dataobject is an “ANY” data object, a matching ACL rule has been found, andthe value “true” is returned, in step 2733. Otherwise, in step 2734, theroutine “authorize 2” obtains the type form the security principle dataobject and, in step 2735, creates a list of all IDs for objects of thistype included in the security context. Then, in the inner for-loop ofsteps 2736-2738, the routine “authorize 2” determines whether any of theIDs in the list match the ID contained in the security principle dataobject. If an ID in the list matches the ID contained in the securityprinciple data object, as determined in step 2737, then the value “true”is returned in step 2733. Otherwise, control flows to step 2729.

FIG. 28 provides a control-flow diagram for the routine “authorizeservices,” discussed above with reference to FIG. 26. In step 2802, theroutine “authorize services” receives a service-definition list,security context, requested URI, and HTTP request furnished as argumentsto the routine, as discussed above with reference to FIG. 26. Then, inthe for-loop of steps 2804-2809, the routine “authorize services” callsthe routine “authorize,” in step 2805, for each service definition inthe list of service definitions. When the routine “authorize” returnsthe value “false,” as determined in step 2806, the routine “authorizeservices” throws an exception, in step 2807 and returns in step 2808.Otherwise, when all of the services in the service-definition list havebeen processed, the routine “authorization services” returns in step2810 without throwing an exception.

Thus, the authorization service provided by the cloud-director backendand the ASMI provided through the cloud-API entrypoint provides aflexible and powerful access-control functionality to the variousservices and functionalities added to the cloud-director-providedfunctionalities and services by the extension facility provided throughthe cloud operator interface via the cloud-API entrypoint. Cloudoperators and other superusers can therefore supplement and evenredefine cloud-director-provided functionalities and services while, atthe same time, associating the functionality and services withaccess-control rules in order that the functionalities and servicesoperate within a multi-tenant VDC environment.

Although the present invention has been described in terms of particularembodiments, it is not intended that the invention be limited to theseembodiments. Modifications within the spirit of the invention will beapparent to those skilled in the art. For example, the authorizationservice can be implemented in many different ways by varying any of manydifferent design, implementation, and deployment parameters, includingthe virtualization layer in which the stretch-deploy operation isimplemented, programming language, control structures, data structures,modular organization, and other such design and implementationparameters. In the above-discussed implementation of the authorizationservice, specific types of data objects and primitives are stored in theauthorization-service database and used to specify ACL rules. Inalternative implementations, other types of data objects may be storedfor the purpose of defining ACL rules. ACL rules may be expressed inmany different types of encodings in alternative implementations.

It is appreciated that the previous description of the disclosedembodiments is provided to enable any person skilled in the art to makeor use the present disclosure. Various modifications to theseembodiments will be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherembodiments without departing from the spirit or scope of thedisclosure. Thus, the present disclosure is not intended to be limitedto the embodiments shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

The invention claimed is:
 1. A cloud-connector subsystem comprising:cloud-connector nodes, each associated with a cloud-computing facility;and a cloud-director server that includes one or more processors, one ormore memories, one or more data-storage devices, and computerinstructions that, when executed on the one or more processors, controlthe cloud-director server to provide, in cooperation with thecloud-connector nodes: a management user interface that provides nativemanagement services and functionalities for creating, administering, andmanaging virtual data centers within one or more cloud-computingfacilities, each associated with a cloud-connector node; an APIentrypoint request/response interface to native services, aservice-extension interface, service extensions created through theservice-extension interface, and anauthorization-service-management-interface; and an authorization servicethat controls access to service extensions created through theservice-extension interface.
 2. The cloud-connector subsystem of claim 1wherein the API entrypoint request/response interface is a hierarchical,URI-based RESTful interface.
 3. The cloud-connector subsystem of claim 1wherein the authorization service comprises: an authorization-servicedatabase; and an authorization routine that is called to determinewhether a request directed to a service extension through the APIentrypoint request/response interface is authorized to access theservice extension.
 4. The cloud-connector subsystem of claim 3 whereinthe authorization-service database stores data objects within one ormore data-storage devices, the data objects including: data objects thateach represents an access-control rule.
 5. The cloud-connector subsystemof claim 4 wherein an access-control rule specifies an authorizationrelationship between one of: a resource class, a resource-class action,an organization, and a user; a resource class, a resource-class action,an organization, and a group of users; a resource class, aresource-class action, an organization, and a right; a resource class, aresource-class action, an organization, and a role; a service, aresource-class action, an organization, and a user; a service, aresource-class action, an organization, and a group of users; a service,a resource-class action, an organization, and a right; and a service, aresource-class action, an organization, and a role.
 6. Thecloud-connector subsystem of claim 5 wherein a resource class is a typeof service; wherein a service is a service extension accessed throughthe API entrypoint request/response interface; wherein an organizationis associated with each virtual data center in a multi-tenantcloud-computing facility; wherein a user is an individual who accesses aservice extension accessed through the API entrypoint request/responseinterface; wherein a group of users is a defined set of users; wherein aright is specific access right; and wherein a role is a job title,professional capacity, or position associated with an individual orgroup of individuals.
 7. The cloud-connector subsystem of claim 6wherein the authorization-service database stores additional types dataobjects of data objects that include: data objects that each represent aresource class; data objects that each represent a resource-classaction; data objects that each represent a service; data objects thateach represent a user; data objects that each represent a group ofusers; data objects that each represent a right; and data objects thateach represent a role.
 8. The cloud-connector subsystem of claim 3wherein the authorization routine returns an indication of whether arequest to a service extension is authorized; and wherein theauthorization routine is supplied sufficient information about therequest to the service extension to allow the authorization routine toextract, from data objects stored within the authorization service,indications of at least a resource class, resource-class action,service, organization, and user, group, right, or role corresponding tothe request to the service extension that the authorization routine thenuses to determine whether or not the resource class, resource-classaction, service, organization, and user, group, right, or role match arelationship defined by an access-control rule.
 9. The cloud-connectorsubsystem of claim 8 wherein the authorization routine retrievesdifferent access-control rules from the authorization-service databaseuntil the most recently retrieved access-control rule matches acombination of primitives selected from among a resource class, aresource-class action, a service, an organization, a user, a group ofusers, a right, and a role corresponding to the request to the serviceextension, in which case the authorization routine returns an indicationthat the request to the service extension is authorized, or until thereare no more different access-control rules to retrieve from theauthorization-service database, in which case the authorization routinereturns an indication that the request to the service extension is notauthorized.
 10. The cloud-connector subsystem of claim 3 wherein theauthorization-service-management-interface is a hierarchical, URI-basedRESTful interface accessed through the API entrypoint request/responseinterface that allows an authorized user to create data objects within,retrieve data objects from, and delete data objects from theauthorization-service database.
 11. A method for extending servicesprovided by cloud-connector nodes, each associated with acloud-computing facility, and a cloud-director server that includes oneor more processors, one or more memories, one or more data-storagedevices, the method comprising: providing an API entrypointrequest/response interface to native services, a service-extensioninterface, service extensions created through the service-extensioninterface, and an authorization-service-management-interface; and anauthorization service that controls access to service extensions createdthrough the service-extension interface; and when a request is receivedthrough the API entrypoint request/response interface and directed to aservice extension, responding to an authorization inquiry, directed tothe authorization service by the service extension, by indicating to theservice extension whether or not the request is authorized.
 12. Themethod of claim 11 wherein the API entrypoint request/response interfaceis a hierarchical, URI-based RESTful interface.
 13. The method of claim11 wherein the authorization service comprises: an authorization-servicedatabase; and an authorization routine that is called to make anauthorization inquiry to determine whether a request directed to aservice extension through the API entrypoint request/response interfaceis authorized to access the service extension.
 14. The method of claim13 wherein the authorization-service database stores data objects withinone or more data-storage devices, the data objects including: dataobjects that each represents an access-control rule.
 15. The method ofclaim 14 wherein an access-control rule specifies an authorizationrelationship between one of: a resource class, a resource-class action,an organization, and a user; a resource class, a resource-class action,an organization, and a group of users; a resource class, aresource-class action, an organization, and a right; a resource class, aresource-class action, an organization, and a role; a service, aresource-class action, an organization, and a user; a service, aresource-class action, an organization, and a group of users; a service,a resource-class action, an organization, and a right; and a service, aresource-class action, an organization, and a role.
 16. The method ofclaim 15 wherein a resource class is a type of service; wherein aservice is a service extension accessed through the API entrypointrequest/response interface; wherein an organization is associated witheach virtual data center in a multi-tenant cloud-computing facility;wherein a user is an individual who accesses a service extensionaccessed through the API entrypoint request/response interface; whereina group of users is a defined set of users; wherein a right is specificaccess right; and wherein a role is a job title, professional capacity,or position associated with an individual or group of individuals. 17.The method of claim 16 wherein the authorization-service database storesadditional types data objects of data objects that include: data objectsthat each represent a resource class; data objects that each represent aresource-class action; data objects that each represent a service; dataobjects that each represent a user; data objects that each represent agroup of users; data objects that each represent a right; and dataobjects that each represent a role.
 18. The method of claim 13 whereinthe authorization routine returns an indication of whether a request toa service extension is authorized; and wherein the authorization routineis supplied sufficient information about the request to the serviceextension to allow the authorization routine to extract, from dataobjects stored within the authorization service, indications of at leasta resource class, resource-class action, service, organization, anduser, group, right, or role corresponding to the request to the serviceextension that the authorization routine then uses to determine whetheror not the resource class, resource-class action, service, organization,and user, group, right, or role match a relationship defined by anaccess-control rule.
 19. The method of claim 18 wherein theauthorization routine retrieves different access-control rules from theauthorization-service database until the most recently retrievedaccess-control rule matches a combination of primitives selected fromamong a resource class, a resource-class action, a service, anorganization, a user, a group of users, a right, and a rolecorresponding to the request to the service extension, in which case theauthorization routine returns an indication that the request to theservice extension is authorized, or until there are no more differentaccess-control rules to retrieve from the authorization-servicedatabase, in which case the authorization routine returns an indicationthat the request to the service extension is not authorized.
 20. Themethod of claim 13 wherein theauthorization-service-management-interface is a hierarchical, URI-basedRESTful interface accessed through the API entrypoint request/responseinterface that allows an authorized user to create data objects within,retrieve data objects from, and delete data objects from theauthorization-service database.